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Abstract 

The BLG model for multiple M2-branes motivates an M5-brane theory with a novel gauge 
symmetry defined by the Nambu-Poisson structure. This Nambu-Poisson gauge symmetry 
for an M5-brane in large C-field background can be matched, on double dimension reduction, 
with the Poisson limit of the noncommutative gauge symmetry for a D4-brane in B-field 
background. Naively, one expects that there should exist a certain deformation of the Nambu- 
Poisson structure to match with the full noncommutative gauge symmetry including higher 
order terms. However, We prove the no-go theorem that there is no way to deform the 
Nambu-Poisson gauge symmetry, even without assuming the existence of a deformation 
of Nambu-Poisson bracket, to match with the noncommutative gauge symmetry in 4+1 
dimensions to all order, regardless of how the double dimension reduction is implemented. 
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1 Introduction 



Recently, significant progress has been achieved in our understanding of multiple M2-branes 
through the BLG model [H |2] and the ABJM model [3]. However, the physics of a system of 
multiple M5-branes still remains mysterious, although the Lagrangian formulation for a single 
M5-brane has been known for over ten years [3]. 

The description of multiple M5-branes is expected to be a non-Abelian generalization of 
the single M5-brane theory, for reasons analogous to why the low energy effective theory for 
TV D-branes is a U(N) gauge theory. Yet a peculiar property of the multiple M5-brane system 
is that its entropy scales as TV 3 , instead of N 2 , for N coincident M5-branes [5] (for large N). 
This suggests that the non-Abelian generalization for M5-branes is not an ordinary Yang-Mills 
theory with the gauge symmetry of a certain semi-simple Lie algebra, for which the dimension of 
the Lie algebra scales as N 2 if the Cartan subalgebra is N dimensional. Thus the first mystery 
about M5-branes is: What is the gauge symmetry for a system of M5-branes? A no-go theorem 
was proven in [BJ, saying that a non-Abelian generalization is impossible. They considered 
general deformations of the theory of N commuting copies of self-dual 2-form gauge theory 
in 6 dimensions, and found that a consistent non-Abelian deformation is always trivial, i.e., 
equivalent to a change of variables. They did not assume the deformed gauge transformations 
to correspond to a particular type of symmetry algebra, but only that the deformation is local. 

Recently, a new Lagrangian formulation of a single M5-brane in a large C-field background 
was introduced [7][5J. □ This theory was derived from the BLG model of multiple membranes 
for which the Lie 3-algebra structure was chosen to be the Nambu-Poisson (NP) structure. The 
gauge symmetry is a non-Abelian generalization of the Abelian gauge symmetry for a self-dual 
2-form gauge potential, with the non-Abelian structure introduced through the Nambu-Poisson 
bracket. We will refer to this theory in the following as the NP M5-brane theory. In contrast 
to the old formulation of M5-brane theory, the usual Abelian gauge symmetry is promoted to 
the non-Abelian gauge symmetry of the volume-preserving diffeomorphisms (VPD) in the NP 
M5 action. The coupling constant for the non-Abelian coupling (the NP bracket) is inversely 
proportional to the C-field background. There is no index going from 1 to N, and thus the NP 
M5-brane theory is not a counter-example of the no-go theorem [6] mentioned above. But it 
reminds us that no-go theorems can always be circumvented. 

Incidentally, the entropy of the system of N M2-branes also exhibit a peculiar scaling be- 
havior as S ~ iV 3 / 2 . It turns out that, at the level of equations of motion (in the absence of an 
invariant inner product), the BLG model can reproduce this scaling behavior using a truncated 
Nambu-Poisson algebra as the Lie 3-algebra [T7j . We take this as a hint suggesting that further 
understanding of the NP M5-brane model may help us to understand this subject better. 

The major evidence that the NP M5-brane action corresponds to a large C-field background 



1 A review on the NP M5-brane is given in [9], and further investigations on this model include [TOl ITT1 IT2l [T3l 

□Him us]. 
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is that it reduces to the D4-brane theory in large .B-field background upon double dimension re- 
duction [8]. A D4-brane in i3-field background is known to be described by the noncommutative 
(NC) Yang-Mills action |181I19|. For large -B-field background, the noncommutativity parameter 
is inversely proportional to the -B-field, i.e., 9 ~ 1/-B, and the Moyal bracket is approximated 
by the Poisson bracket at the lowest order in the ^-expansion 

[f(x),g(x)\* = f * g - g * / = 0'>i),fi):g + O(0 3 ). (1.1) 

The double dimension reduction of the NP M5-brane captures the Poisson bracket structure of 
the NC D4-brane action, but misses all the higher order terms in the ^-expansion. This means 
that the NP M5-brane is just a lowest order approximation for large C-field background in the 
1/C-expansion. The natural question is thus: Can we deform the NP M5-brane theory such that 
its double dimension reduction gives the NC Yang-Mills theory to all orders in 9? 
The aim of this paper is to ask a more fundamental question instead: 

Can we deform the NP gauge symmetry such that it reduces to the NC gauge sym- 
metry (including higher order terms) upon double dimension reduction? 

Obviously, if the answer to this question is negative, the answer to the former question must 
be, too. Notice that this latter question can be asked without referring to any specific model 
of M5-brane theory. As D4-branes are physically the same as M5-branes wrapping on a circle, 
we are tempted to assume it possible to "lift" the gauge symmetry on D4-branes to the gauge 
symmetry on M5-branes. Since the Poisson limit (the lowest order approximation in the small 
^-expansion) of the NC gauge symmetry can be lifted to the NP gauge symmetry, it is natural 
to expect that the NC gauge symmetry can be lifted to a certain deformation of the NP gauge 
symmetry. However, we will prove that the answer to the question above is no. 

2 Review of NP Gauge Symmetry 

The Nambu- Poisson bracket is a natural generalization of the Poisson bracket 
bracket 

{f,g} = j j difd jg = 1,2) 

can be used to generate the area-preserving diffeomorphisms in 2D by 

6 A </> = {A,<I>}, (2.2) 

the NP bracket 

{A, B, C} = e abc d a A d b B d c C (a, b, c = 1, 2, 3) (2.3) 

2 On the other hand, the NC D4-brane action can be directly obtained from the BLG model by choosing a Lie 
3-algebra which does not resemble the NP algebra |20| . 



. As the Poisson 
(2.1) 
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can be used to generate 3D volume-preserving diffeomorphism (VPD) by 



6 (A ,B)<i> = {A,B, ( f>} (2.4) 

for two arbitrary functions A,B. As we can linearly superpose transformations to obtain new 
transformations, a generic 3D transformation is of the form 

5 K <f> = gK a d a cf>, (2.5) 
where K a are arbitrary functions satisfying the constraint 

d a K a = 0. (2.6) 
We can compute the Lie bracket of the 3D VPD 

[S K ,S K '] = S K u, (2.7) 

where k" is found to be 



K " a 



-g [K b d b K ,a - K lb d b K a ^ , (2i 



which satisfies the condition (|2.6p as required. 

It was shown [8] that, via double dimension reduction, the NP M5-brane action reduces to 
the Poisson limit of the NC super Yang-Mills action on 4+1 dimensions. Essentially this is 
because of the relation 

{A,B,x 3 } = {A,B}, (2.9) 

where the left hand side is the NP bracket, and the right hand side is the Poisson bracket. As 
the Poisson structure only accounts for the lowest order terms in the Moyal bracket defined for 
a noncommutative space 

[A{x),B{x% =A*B-B*A = i9e ij diAdjB + 0(# 3 ), (2.10) 



where the Moyal product * is defined by 
A{x)*B{x) = e^ 6elJ ^^ J A(x)B(y) 



= A(x)B{x) + -9{A{x), B(x)} + 0(6 2 ), (2.11) 

y=x 



we are looking for ways to recover the higher order terms in 9 in order to obtain the full NC 
D4-brane action. 

There are several logical possibilities how a full NC D4-brane action can be obtained from 
a double dimension reduction of the M5-brane theory, depending on the extent to which a 
deformation is needed. The followings are the logical possibilities. 

1. The NP M5-brane theory does not have to be deformed; only the ansatz for double di- 
mension reduction used in |8| should be deformed. 
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2. The NP bracket has to be deformed so that the NP gauge symmetry is deformed accord- 
ingly. 

3. A more drastic deformation of the gauge symmetry is in need, so that the notion of NP 
bracket has to be forsaken. 

Let us consider each possibility in more detail in turn. 

3 Deformation of Double Dimension Reduction 

The first possibility asks for the minimal deformation. Although it will be hard to convince ev- 
eryone that one has exhausted all possible ansatz for double dimension reduction, it is actually 
possible to rule out the first possibility without any assumption about the double dimension 
reduction. The reason is as follows. Regardless of how double dimension reduction is imple- 
mented, if it is possible to obtain NC gauge symmetry from the gauge symmetry of 3D VPD via 
some sort of double dimension reduction, it must be possible to embed the NC gauge symmetry 
as a subgroup of the 3D VPD, because double dimension reduction is always a restriction of the 
gauge symmetry to a subgroup. Let us now prove that it is indeed impossible to embed 2D NC 
gauge symmetry in 3D VPD. 

Identifying NC gauge symmetry as a subgroup of 3D VPD means that there exists a map 
k(X), which maps a NC gauge transformation parameter A to a VPD transformation parameter 
k. We already know that the Poisson limit of the NC gauge symmetry is a subgroup of the 3D 
VPD. The embedding map is 



k£(\) = { T (3-1) 

where \i = 1, 2, 3, i,j = 1, 2, and we need to identify 

9 = 0. (3.2) 
To embed the NC gauge symmetry in 3D VPD, we need a deformation of the map above 

k(X) = k (X) + 0«i(A) + 9 2 K 2 (X) + • • • , (3.3) 
such that for arbitrary A, A', 

«M([ Aj X %) = -g (^(A)d^(A') - k"(A')0„k"(A)) , (3.4) 
where g can be related to 8 through a deformation of (|3.2p 

g = e + h x e 2 + h 2 e 3 + • • • . (3.5) 

We will now prove that it is impossible to find K\ and k 2 such that (|3.4p holds for all A, A'. 
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The NC gauge symmetry algebra is denned by the Moyal bracket (|2.10p . and can be expressed 
as a ^-expansion 

[A, A']* = 8{X, A'} + 6 2 M 1 (X, A') + 6 3 M 2 (X, A') + • • • , (3.6) 
where Mi (A, A') = and 

M 2 (A, A') = ^e kl e mn (d l d k d m X)(d J d l d n X'). (3.7) 

In general, the maps K n (X) is linear in A, and in principle they can depend on the gauge 
potential or other fields of the gauge theory. However, the closure relation (|3.4p should be 
satisfied regardless of the values of these fields, thus we can set all other fields to zero and 
consider pseudodifferential operators on A as 



K 'n 



oo 

^(A) = ^iT^™(x)(^ 1 ---^ fc A). (3.8) 

k=0 

Let us now try to solve K n order by order from f|3.4[) . The lowest order terms are 

<({A, A'}) = - («S(A)a,«ff (AO - «g(A0ft,^(A)) . (3.9) 



This identity is already satisfied by our choice of kq (|3.ip . 
At the next order, f|3.4|) gives 



<({A, A'}) = h lK %({X, A'}) - (^(A)c^(A') + <(A)&<(A') - (A o A')) . (3.10) 
Expanding K\ as 

oo 

^{x) = Y,K w "" k ^-, k \ (3.H) 

k=0 

we notice that only F$, and will be relevant for the argument below. 

A necessary condition for (|3.10p to hold is that all the terms of the form d 3 Xd 2 X' cancel. 
Restricting A and A' to be independent of x 3 , we then find 

3F$ jkl {djd k X, d t X'} = -e^F^dkdtdmXdjdnX'. (3.12) 

For A = e tk ' x and A' = e lk '' x , this implies that 

k'iF^^kjhkl = 0, (3.13) 

which is solved by 

F l 3 ljhl kjk k = ae il (3.14) 

where a can be an arbitrary function. However, by definition F^ kl must be symmetric with 
respect to (j,k,l), and since there is no way to preserve the relation above while symmetrizing 
j,k,l, the only possibility is that F^ kl = 0. 



3 If F'^ kl and p^ klm obeys (|3.14p and (|3.19p respectively, F^ kL and p^ klm must both vanish because 

T-.i\jkl i-.ZHfcz T-^iljifc 1 - l k\iil r-.kljli r-.iiilk r^iljkl / 1 ~\ 

^3 =-^ 3 =-F :i =F 3 = F 3 =~F 3 =-F 3 , (3-15) 
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Similarly, for F4, we consider terms of the form d 4 Xd 2 X', 

± F m™ {djdkdlX ,d m \'} = -^F^dkd^dnXdjdrX'. (3.17) 
For A = e lkx and A' = e tk x , this implies 

klF^kjhkk^ = (3.18) 

which is solved by 

Ff iklm k j k k k l = ^ m , (3.19) 

'4 



where (3 is just a constant. Since p % \i klm \ s totally symmetrized among jklm by definition, ()3.19|) 



requires that p^ klm = q identically. 
At order 9 4 , (|3.4|) gives 

4({A, A'}) = -<(M 2 (A, A')) + h 2 K%({X, A'}) - h x («£(A)d„<(A') + k?(A)$,«£(A') - (A o A')) 
- «(A)^<(A') + K%(X)d v K%(X') + ^(A)^<(A') -(Ao A')) . (3.20) 

Let us now check whether non-trivial K2 exists. Consider the case li = i. The first term on the 
right hand side is of the form <9(<9 3 A(9 3 A'). This term has no match on the right hand side, as 
each term on the right hand side has less than 3 derivatives on either A or A'. A priori such a 
term could arise from the term K 1 {{X)d l ,K^{X'), but this can not happen as we have proven above 
that F^ kl = p^ klm = 0. As a result, the first term on the right hand side has to be cancelled 
by the term on the left hand side. For the special case 

A = e ik ' x , X' = e ik '- x , (3.21) 

the 1st term on the right hand side and the term on the left hand side of (|3.20p are 

- i— e ij {k + k')j(k x k'fe i{ - k+k > x vs g(k x k')4(e^ k+k '> x ). (3.22) 
3 

Apparently K2 has to be a pseudo-differential operator, and K?>( e ^ fc+fc '' ) x ) = + k')e l ( k+k '^' x 
for some function f % . After a change of variables 

p = k + k', q = (k' — k)/2, (3.23) 

the two terms become 

-4^(px?fe^ vs g(pxq)f{py p - x , (3.24) 

hence we need 

f(p)=i^ Pj (pxq) 2 (3.25) 

for ()3.20p to hold. Yet this is impossible because f l (p) is independent of q. Therefore K2 can 
not exist. The conclusion is that NC gauge symmetry can not be a subgroup of 3D VPD. 



r^iljklm r^mljkii r^mljkil j-^l\jkim r-^l\jkmi j-*i\jkml r-^i\jklm / i p\ 

t i — —r A — —r A = m< 4 — t A — —r 4 — —t A . (6. Lb) 



6 



4 Deformation of the NP Bracket 



The next simplest possibility of obtaining a new theory for finite C-field background is that there 
exists a deformation of the Nambu-Poisson bracket analogous to the Moyal bracket, which can 
be viewed as a deformation of the Poisson bracket. In order for the deformed Nambu-Poisson 
bracket to be ready to define a gauge symmetry, the new bracket must satisfy the fundamental 
identity 

[A,B, [C,D,E]] = [[A,B,C],D,E} + [C,[A,B,D},E} + [C,D,[A,B,E]}. (4.1) 
In terms of the definition of a transformation generated by the deformed NP-bracket 

6 {A)B) = [A,B,-}, (4.2) 
the fundamental identity can be understood as the covariance of the bracket 

5 {AB) [C,D,E] = [8 {AB) C,D,E] + [C,S (A>B) D,E] + [C, D,5 (AB) E]. (4.3) 
It can also be rewritten as the closure of the algebra of transformations as 

[ti(A,B),$(C,D)}E = S ([A,B,C],D)+(C,[A,B,D])E. (4.4) 

This identity is usually referred to as the fundamental identity. If the 3-bracket [A, B, C] is 
assumed to be totally antisymmetric, the algebra satisfying (|4.4p is called a Lie 3-algebra, and it 
can be used for the Af = 8 BLG model. If the 3-bracket [A, B, C] is antisymmetric only in A, B, 
the algebra satisfying (|4.4p can be used to define an M = 6 model in 3D [21] . In the literature 
[22 , , a deformation of the NP bracket often assumes the Leibniz rule 

[A, B, CD] = [A, B, C]D + C[A, B, D]. (4.5) 

But it turns out that the 2nd possibility, which is the theme of this section, can be ruled out 
even without assuming the Leibniz rule. Actually, we will see in the next section that, without 
any assumption about the form of the gauge symmetry except that it is a deformation of the 
3D VPD (the 3rd possibility discussed at the end of sec. [2]), it is impossible to obtain NC gauge 
symmetry upon double dimension reduction of any sort. Hence the possibility that a deformed 
NP bracket reduces to the Moyal bracket is also ruled out. 

5 Deformation of the Gauge Symmetry 

In general, it is interesting to know whether the algebra of d dimensional VPD admits any 
nontrivial deformations. As any algebra of symmetry with infinitesimal transformation laws, 
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the deformed symmetry must have the structure of a Lie algebra. In this case, we are looking 
for an infinite dimensional Lie algebra satisfying the Jacobi identity 

[[X,Y],Z} + cyclic = [[X,Y],Z} + [[Y,Z},X} + [[Z,X],Y}=0, (5.1) 

as a deformation of the 3 dimensional VPD. 

Assuming that there is a deformation of the Lie bracket [•, -]o of a Lie algebra G, one can 
expand the deformed Lie bracket as 

[X,Y] g = [X,Y]o+gc l (X,Y)+g 2 c 2 (X,Y) + --- , (5.2) 

where g is the deformation parameter and ci,p2,--- are bilinear functions on Q. The Jacobi 
identity has to be satisfied order by order. While it is trivial at the 0-th order, the Jacobi 
identity at the 1st order is 

6cx(X,Y,Z) = [c 1 {X,Y),Z] + c 1 ([X,Y]o,Z)+ cyclic = 0. (5.3) 

Note that the 1st order deformation would be considered trivial if it merely corresponds to 
a change of basis of the Lie algebra generators. That is, a linear transformation on Q 

A^<D(A) = X + gb(X) + 0(g 2 ), (5.4) 

where b is a linear map on G, is not considered to change the Lie algebra, although it induces a 
change of the appearance of the Lie bracket 

MX), $(Y)] = $([X, Y]o) + g5b(X, Y) + 0(g 2 ), (5.5) 

where 

Sb(X, Y) = [b(X), Y] + [X, b(Y)} - b([X, Y] ). (5.6) 

Therefore, if c\ = 5b for some b, c\ is considered a trivial deformation. Furthermore, if two 
choices of c\ are considered equivalent if their difference is 5b for some b. In short, c\ should 
be considered as an element in the cohomology defined by 5, denoted by H 2 {Q,Q). A necessary 
condition for the existence of nontrivial deformations of Q is then the cohomology class H 2 (Q, Q) 
is nontrivial. 

In |24j . it was shown that H 2 (Q, Q) is trivial for Q being the VPD on compact manifolds in 4 
or higher dimensions. For VPD on 3 dimensional compact manifolds, H 2 {Q, Q) is 1 dimensional. 
However, even if H 2 (Q, Q) is nontrivial, it does not imply the existence of nontrivial deformation 
of Q. One still needs to check that the Jacobi identity holds at higher orders. At the 2nd order, 
the Jacobi identity implies that 

- [ci (ci (A, Y), Z) + cyclic] = 5c 2 (X,Y,Z). (5.7) 
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Thus a nontrivial deformation of the 3D VPD exists only if the left hand side of the equation 
above is a trivial element in H^{Q,Q) for the nontrivial element c\ (the only one) in H 2 (Q,Q). 
This turns out to be not true, and the conclusion is that the VPD is rigid for all d > 3, i.e., it 
does not admit any nontrivial deformation [21]. Q 

The rigidity of the gauge symmetry of 3D VPD implies that there is no nontrivial deformation 
of the Nambu-Poisson bracket. 

Strictly speaking, this theorem only applies to compact manifolds, % that is, to the case when 
the M5-brane worldvolume is a product space M1+2 x A3 where A3, the 3D space on which 
the Nambu-Poisson bracket is defined, is compact. It is possible that a noncompact A3 can 
still admit a deformation of its VPD. However, in a physical theory, finite-energy physical states 
are local configurations which decay to zero at infinity. In this sense, the noncompact space 
M 3 should be treated the same as the infinite- volume limit of, say, S s or T 3 . Thus we expect 
that the no-go theorem also applies to M5-branes with non-compact base space, at least when 
certain locality conditions are imposed. 



6 Conclusion 

We have proven in the above the no-go theorem that it is impossible to deform the NP gauge 
symmetry so that it includes the NC gauge symmetry as a subgroup to all orders. The basic 
reason of this no-go theorem is that the NP gauge symmetry (for compact A3) is precisely the 
3 dimensional volume-preserving-diffeomorphism (VPD). While the 3D VPD does not include 
the NC gauge symmetry in 2D as a subgroup. It is also rigid against deformation. 

The rigidity of the 3D VPD implies that the NP M5-brane theory is also rigid, in the 
sense that it can only be deformed by adding gauge invariant terms to its Lagrangian without 
deforming its gauge transformation laws. This means that we can not directly identify the gauge 
symmetry of an NC D4-brane with the NP gauge symmetry of an M5-brane, regardless of how 
we deform the NP M5-brane theory. 

On the other hand, the no-go theorem does not imply that it is impossible to construct an 
M5-brane theory that reduces to the D4-brane theory in i?-field background. In order to find 
the correction terms in the NP M5-brane theory to reproduce NC D4-brane theory, one can 
first use a Seiberg-Witten map to rewrite the NC D4-brane action in terms of the gauge fields 
with Poisson gauge symmetry, and then the gauge transformation laws of the D4 and M5-brane 
theories can be directly identified through the double dimension reduction used in [5]. The 
correction terms will be easy to find, at least order by order, after the gauge symmetries are 
matched. 

4 2D VPD admits deformation, e.g. the noncommutative gauge symmetry generated via the Moyal bracket. 
In [23] a quantum version of the Nambu-Poisson bracket is constructed based on the so-called Zariski algebra 
defined for R 3 . It remains to be checked whether this algebra leads to a deformation of VPD. If it does, it provides 
an example of deformed 3D VPD for noncompact space. 
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The moral of the no-go theorem presented in this paper is that there is still a lot to be 
learned about the M5-brane physics, even before we take up the challenge of the problem of 
multiple M5-branes. 
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